DOCUMENT RESUME 



ED 423 273 



TM 029 078 



AUTHOR 
TITLE 
PUB DATE 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



Dirks, Matthew 

How Is Assessment Being Done in Distance Learning? 

1998-00-00 

27p.; Paper presented at the NAU/web.98 conference 
(Flagstaff, AZ, May 28-30, 1998) . 

Reports - Research (143) -- Speeches/Meeting Papers (150) -- 

Tests/Questionnaires (160) 

MF01/PC02 Plus Postage . 

Case Studies; *Cheating; *College Faculty; *Distance 
Education; Educational Assessment; *Electronic Mail; 
Feedback; Higher Education; Interactive Television; Research 
Methodology; Student Evaluation; *Test Construction; Test 
Use 



ABSTRACT 



This report is a summary of research that was recently 
conducted with instructors of distance learning programs. Sixty-two 
professors from 10 categories of courses completed an e-mail survey. The 
advantages and disadvantages of assessments seemed to revolve around the key 
issues of time, synthesis or knowledge acquisition, subjective or objective 
grading, and the real world. The four types of assessments most commonly 
mentioned were case studies, examinations, papers, and projects. The power of 
case studies was identified as their current, real-world application of 
concepts. They were used for improvement feedback, to evaluate, to facilitate 
learning, and to measure application and understanding. Examinations, the 
most traditional form of assessment, were cited for feedback and grading. 
Papers were most commonly used for the detailed analysis of one subject area, 
while projects were used by the instructor to help students set goals, 
evaluate the instructor, and provide grades. Quality was being maintained by 
these professors in generally informal ways, with comparisons and reviews of 
the assessments themselves being common approaches. In these distance 
learning situations, e-mail was the most common way students submitted work, 
although students used site facilitators or dropped off work when possible. 
Users were divided about the advantages and disadvantages of different 
delivery methods for lessons and assessment. Text delivery of content was a 
difficult area to define, but the typical course mailed a syllabus, learning 
resources, and text books to the student, who turned in assignments by 
e-mail, the Internet or fax. Cheating on assessments was the most emotionally 
charged issue in the study, with even defining cheating being difficult at 
the distance learning level. The first appendix discuses the study 
methodology and the e-mail survey, the second presents the interview guide, 
and the third is the survey itself. (SLD) 
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Assessment in Distance Learning 



This report is a summary of research that was recently conducted with instructors 
of distance learning MBA programs. Instead of providing tables and tables of data 
emphasis will be placed on current practices in an attempt to help decision makers and 
instructors identify practices they want to emulate. With a clear understanding of current 
practices it is hoped that the next generation of distance learning programs can improve 
upon what is currently being done. 

One of the greatest difficulties is sifting through the research and making 
decisions on what others might want to know. Questions are going to be presented to 
help organize the data in a meaningful way. Hopefully, these questions are relevant and 
the answers to them will help those involved in distance learning. The questions that will 
be addressed in this paper are: 

• What are the advantages and disadvantages of the different types of 
assessment and how are they being used? 

• How much time is being spent doing assessment related activities? 

• How important are exams? 

• How is quality being maintained? 

• How distance learning impact the communication between the learner and the 
instructor? 

• What are the advantages and disadvantages of the different delivery media and 
how does delivery impact student to student communication? 

• Are distance learning instructors spending more time with on or off campus 
students? 

• How is cheating being handled in the distance learning environment? 

Appendix A will address the sample and the data collection techniques. If any of the 
questions seem relevant it is recommended that you review the Appendix to ensure 
everything is understood in context. Also consider that this information has been 
summarized a great deal and in that process inaccuracies may have been introduced. 
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What are the advantages and disadvantages of the different types of 
assessment and how are they being used? 

The advantages and disadvantages of assessments seemed to revolve aroimd five 
key issues: time, synthesis or knowledge acquisition, subjective or objective grading, and 
real world. You will notice that these issues boimce back and forth between the 
advantage and disadvantage column depending on the assessment used. 

Instructors were asked why they used assessments in the phone interview 
(Appendix B) and what purpose their assessments serve in the written survey (Appendix 
C). The answers were closely tied to the advantages and disadvantages. The primary 
reasons given for having assessments included providing feedback, giving grades, and 
motivation. The four types of assessment that were most commonly mentioned are: case 
studies, exams, papers, and projects. 

Case Studies 

The results indicated that case studies power is in current, real world application 
of concepts. Here are the primary advantages and disadvantages mentioned by the 



participants: 






Case Studies 


Advantages 


Disadvantages 




• Requires reading and comprehension 


• Subjective grading 




• Demonstrate ability to apply 


• Time (grading and doing) 




• Opportimity for thorough feedback 

• Very current 


• No right answers 




• Real world 





Case studies were used for improvement feedback, to evaluate, to facilitate learning, and 
to measure application and imderstanding. The impact this has had on learning included 
more reading, more work, and encouraged critical thinking through the synthesis of 
information. 
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Exams 

Exams represent the most traditional form of assessment. There were several 
comments indicating that professors didn’t know why they had them, they were only 
there to give grades, or it is part of the learning process. Even though their use can be 
based on tradition, they have a unique set of advantages and disadvantages: 



Exams 


Advantages 


Disadvantages 




• Objective grading (no 


• Difficult to grade (subjective) 




debates) 


• Not enough writing 




• Motivational 


• Selective sampling 




• Very efficient 


• Not tie to the real world 




• Cover many topics 


• Not comprehensive 




• Quick turn around 


• Can’t ask follow up questions 



The purpose of exams was two fold: feedback and grading. Exams impact on learning 
included motivation and improved learning. Exams encompasses final exams, quizzes, 
essay test, and multiple choice tests because of this we see tests as both objective and 
subjective. Exams seem to be used a lot simply out of tradition, but they are very useful 
because they motivate students, can provide quick feedback, and tend to be more 
objective to grade. Exams were used extensively and the question on “How important are 
exams?” addressed how much weight was put on exams in determining the final grade. 

Papers 



Papers are commonly used for an detailed analysis of one subject area. The 
advantages and disadvantages are: 



Papers 


Advantages 


Disadvantages 




• Tailored to student needs 


• Subjective (low retest 




• Real world (can’t fake it) 


reliability) 




• Requires synthesis 


• Time to grade 




• Have to learn to write 


• Can’s see actual 




• A lot of personal feedback 


performance 
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I I • Cover’s only one subject in I 

I I depth I 

••••••••.! I } 

Like exams, papers were used to provide feedback and give grades. The impact of papers 
was reported as improved integration skills, writing skills and application. 

Project 



Projects include marketing projects, spread sheet projects, and project reports. 
The advantages and disadvantages of projects mentioned by the participants were: 



Projects 


Advantages 


Disadvantages 




• A lot of flexibility 


• Time (feedback, comments) 




• Solve real problems 


• Non-self motivated students 




• Requires critical thinking 


flounder 




• Longer assessments gives better 


• Subjective 




insight into learning 


• Students worry about a bad 




• Synthesis information 


grade 



Projects were used to help the students set goals, evaluate the instructor, and provide 
grades. Projects were used to impact students writing ability and creativity. 



The professors interviewed indicated that they made their assessment choices 
based on their goals, the tools capabilities, and the circumstances of the class. Instructors 
working in distance learning are not creating new forms of assessment and they most 
frequently choose assessment models that they have used before or have seen used. 



How much time is being spent doing assessment related activities? 

In the written e-mail survey (Appendix C) instructors were asked: 

• Considering all the time you spend on this course (100%), how much of that 
time do you think you spend on student assessment (writing tests, grading, 
giving feedback, and reporting scores)? 
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The responses indicated that the newer teachers (2-9 years teaching) spent over half their 
time on assessment (56%), those with 10-19 years experience spent less time on 
assessment (46%), and the career instructors (20+ years teaching) spent the least amount 
of time doing assessment (35%). The data is very different when you look at experience 
teaching the given coxorse. Teachers who had only been teaching the course for a year or 
less spent the least amount of time on assessment (40%), teachers who had taught the 
course for 2 to 4 years spent half their time on assessment (51%), and instructors who had 
been teaching the course for over 5 years spent 49% of their time doing assessment. This 
seems logical with first year teachers having to develop course materials and more 
experienced instructors managing the assessments and feedback more effectively. 

The table below represents what percentage of time instructors teaching small, 
medium and large classes were spending on assessment. The table shows that the small 
classes spent considerably more time on assessment then the larger classes. The data may 
have been influenced by the number of correspondence instructors that participated in the 
research, but a chi square indicated that the variables are associated (p>.01). The last 
column indicates the average amount of time spent on assessment for that class size. In 
other words small class instructors on averaged spent 64% of their time doing assessment 
related activities versus large class instructors which on average spent 37% of their time. 

Class Size vs. Percent Time Spent on Assessment 



Class Size 


Small % time 
on 


Medium % 
time on 


Large % 
time on 


Average % 




Assessment 


Assessment 


Assessment 


time on 




(1% - 33%) 


(34% - 66%) 


(67% - 99%) 


Assessment 


Small Class 
(1-15 students) 


6 


1 


13 


64.1% 


Medium Class 
(16-44 students) 


13 


4 


4 


39.6% 


Large Class 
(45+ students) 


12 


6 


2 


36.8% 
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How important are exams? 

Exam usage varied by subject matter and by class size. Self reporting and 
participants course syllabi were used to determine how much of the grade was determined 
by exams. Subjects like finance and economics consistently relied on exams to determine 
more than 50 percent of the grades. On the other hand, subjects like ethics, human 
resources management, and quality management very rarely use exams to determine more 
than 50 percent of the grade. 

The table below indicates how significant exams were compared to class size. As 
expected, exams were a larger portion of the grade as the class size increased. The small 
classes and large classes have reversed frequencies in terms of small percentage of grade 
in exams versus large percentage of grade in exams. The fourth column was used to 
indicate what percentage of the respondents in each category used exams to determine 
more than half of the grade. In the small class group on 19% of the participants used 
exams to determine more than half the grade and in the large group 64% used exams to 
determine more than half the grade. This indicates that exams are much more heavily 
relied upon in larger classes. 

Class Size vs. Percent Grade Determined by Exams 

Class Size Small % of Medium % of Large % of % that have Vi or 
Grade in Exams Grade in Exams Grade in Exams more determined 
(1%-33%) (34%-66%) (67%-99%) by Exams 

Small Class 7 1 4 19% 

(1-15 students) 

Medium Class 2 9 5 50% 

(16-44 students) 

Large Class 4 9 7 64% 

(45+ students) 



How is quaiity being maintained? 

Quality has increased importance in distance learning because there is increased 
anxiety concerning whether the same level of education is being delivered. There are 
four many areas of focus for quality control in the distance learning MBA programs that 
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were surveyed: informal systems, measures on assessments, application, and university 
support. 

Informal Systems 

The first type of informal quality check is in comparisons. Comparisons are being 
made between students by checking the spread in grades, comparing one semesters grades 
to the next semester, and comparing on versus off campus students. Instructors indicate 
that these systems give them a feel for how well the assessments are differentiating 
students. These are informal systems because there aren’t any measures taken, they 
aren’t done systematically, and they are used sporadically when the instructor is 
interested or has time. 

The major type of informal system is to rely on experience. Many professors 
indicated that they had been teaching for years and they know what a good assessment is 
and how well it will work with their students. This was also tied to face validity. The 
more experienced instructors seemed less concerned with quality because they knew what 
to ask and how to assess students abilities. 

Another type of informal system was to relate student performance to an outside 
measure. CPA scores were sometimes collected by the instructor to find out if the 
students were getting the information they needed. This is considered an informal system 
because the professor may only contact a few students and received the information 
through verbal reporting. One instructor indicated that job placement rates and 
employment rates after completing the degree were meaningless because most of her 
students were employed when they came into the program and their companies were 
paying for the education. She thought the employment rate pre and post the MBA were 
probably equivalent, but over the years their salaries would increase. 

Measures on Assessments 

The most common form of quality checks occurred with the assessments 
themselves. Instructors indicated that they used question banks, checked internal 
reliability, through out questions everyone missed or got, and used old exam questions. 
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A few professors indicated that the internal reliability was automatically provided with 
their tests results and they were using them for quality checks. 

Another form of quality checks concerning the assessment revolved around the 
process and procedures used to grade the assessments. Instructors indicated that they 
used universal grading sheets that clearly indicated what should and should not be 
marked correct. Other instructors indicated they only grade one question at a time so 
each student was held up to the same criteria. Only a few instructors indicated they went 
back through their assessments to determine if the questions actually matched the 
objectives that they had established for the course. 

Application 

The practical side of quality is application. Many instructors indicated that the 
assessments they choose were working because they knew by the end of the course that 
the student could do it. Instructors commonly mentioned the change in ability level of 
the students as their evidence that the assessments were working. One instructor put it 
this way, “I am preparing people to work in the real world, if they can do that when I am 
done, then I have been successful.” 

University Support 

Some of the emerging distance learning programs are making giant strides in the 
area of quality control. The issue was clarified when one of the correspondence 
instructors indicated how difficult it was to be fair and consistent across students. “Every 
week there are new students beginning the course, and it is difficult to make sure that you 
always grade them the same way.” The universities are aware of this and they have put in 
systems to help the instructors maintain quality and consistency. There are currently 
mentoring programs with senior faculty and some universities support faculty audits. A 
new program at the University of Phoenix randomly selects students in the distance 
learning program for a review. All of the students papers and the instructors comments 
are reviewed to give the instructor guidance on grading and providing feedback. Another 
leap forward has been in the area of curriculum quality teams. These teams meet every 
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six months to determine if the course, or its assessments, need to be revised. This helps 
to ensure that the course is on track and current. These systems have been added to the 
other more common types of quality control measures used by universities such as 
training and professional work shops 

Quality control measures are becoming more and more important. Quality seems 
to be one of the areas that some of the emerging programs are investing in that could 
produce great results. Many universities policies seem to get in the way of quality 
assessments. Several of the respondents indicated that they used exams because they 
only had 72 hours to turn in grades. It seems that universities may need to provide a 
different level of support based on whether assessments are used to ascertain the students 
abilities or if they are used to determine grades, differentiate students, or motivate 
students to work. 

How does distance learning impact the communication between the learner 
and the instructor? 

Instructors were asked how students submit their work, contact the professor, and 
how they respond to questions. It became apparent that the students and the professors 
took advantage of the most convenient communication tool available to meet their needs. 
Most of the communication between the instructor and the learner does not take place 
face to face and is done via some form communication technology. 

The most common way students submit work is via e-mail. E-mail was 
mentioned as the most common way of submitting work regardless of the delivery 
medium. When possible, students also use site facilitators or drop off assignments in 
person. Students also use fax, the Internet, and regular mail. One instructor had, 
disallowed the use of e-mail due to the amount of computer viruses he was receiving 
from his students. 

When students have questions the most common ways they contact the professor 
is by e-mail or phone. Many professors indicated that student questions on e-mail are 
much more informal and usually not as well thought out. Professors seemed to appreciate 
e-mail questions because it gave them time to consider their responses. Phone calls were 
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not as well appreciated because it was usually difficult to catch the other party and phone 
tag can get old quickly. The advantage of a phoned in question is that the student can 
clarify their question and pursue issues in a one on one environment. Several professors 
indicated that they “had no life” because they tried to be available for student phone calls 
and that using the phone to teach is not efficient because they end up re-teaching the same 
content several times. 

Professors most commonly respond to questions in the same medium that the 
question was received. Professors indicated that it was very rare to respond to an e-mail 
with a phone call, but they had answered some questions during their presentation or 
lecture so that the entire class could benefit from the response. The same medium was 
also used when returning students work. If materials are handed into a site coordinator, 
the site coordinator commonly returns them. The final assessment was an exception 
because, if they are going to be returned, they are commonly mailed back to the student. 
Professors also indicated that in some instances where papers, and other assignments, 
were turned in via e-mail they had printed them out and made comments in long hand 
before returning them by regular mail. 

What are the advantages and disadvantages of the different delivery media 
and how does the delivery impact student to student interaction? 

Delivery media can be compared on several different levels. Ease of operation, 
cost, and availability are some important variables and are covered in other studies. This 
study looked into how the different technologies were being used and how they impact 
instruction. The six different delivery media included in the study are; 2-way video, 
audio, broadcast video, Internet, text, and video tape. The respondents were divided 
based on the primary way that the content was delivered to the learner. The primary 
issues that were repeated were: lack of face to face communication, the problems with 
technology, and the fact that asynchronous instruction commonly means that the 
instructor is always on duty. 
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2 way Video 

2 way video offers some advantages in communication. It is one of the few 
distance learning technologies that allows dynamic real time interaction. Instructors 
using 2 way video had the largest average class size (48.3 students per class) and the 
largest class in the survey (180 students) was taught using this technology. 

Instructors indicated that they liked the flexibility and the quick communication. 
They also indicated that they used more technology when teach this way (Elmo and 
overheads) and its biggest advantage was its ability to reach isolated students. 

The primary disadvantages included technology problems and discomfort with the 
distance. Technology problems included echoes in the audio, only seeing one site at a 
time, and not being able to see all the students at a site. The students also showed a 
discomfort with the technology by being apprehensive and uncomfortable around the 
microphones or cameras. 

Student to student communication occurred frequently with this kind of delivery. 
The students at the different sites naturally had discussions and met for study groups. 
Instructors encouraged this by requiring group projects, group presentations, and site 
teams. It was more difficult to get students to communicate between sites. One 
instructor recognized this problem and required each team to have at least one member be 
present at the originating site. This ensured that each team had access to the instructor 
and that the students at the originating site didn’t have any advantages over the distance 
learning students. 

Audio 

Audio delivery is being used extensively by Regent’s University and they were 
the only university in this study using audio as the primary way to deliver content. One 
professor indicated that this was the best way to reach their students because most of their 
students had to commute and they could replay the tapes as they were sitting in the car on 
the way to work. The average class size for audio deliver was 3 1 students. 

The advantage of this delivery system is that it is totally asynchronous. This 
means that the student can listen to the content when ever they have time. They used e- 
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mail as the primary form of interaction and gave students very quick feedback to their 
questions and on their assignments. The disadvantage of this system was the reduced 
quality of communication. The instructors rarely got to know the students, not even their 
voices. There was less immediacy and the communication was less personal. 

Students were encouraged to participate in discussion groups on the web and there 
was group project assignments. These activities allowed the students to get together and 
share ideas, but the instructor wasn’t present. 

Broadcast Video 

Broadcast video utilizes current television technology. This was usually 
augmented by phone in lines, e-mail and text. The breadth of this delivery medium was 
illustrated by one professors comment, “I figure only about one third of the calls I receive 
during my class are from students, the rest are from the general public simply interested 
in the subject matter or looking for help with their own management problems.” 
Broadcast video had the second largest average class size, 47 students. It was used to 
teach classes of 9 to 143 students. 

Like 2 way video, the biggest advantage of broadcast video is the large number of 
students that can be reached. The disadvantages concern the lose of immediate 
communication. Phone in systems work well, but by the time the student was able to call 
the professor may have already moved onto another topic. With the way the broadcast 
video is currently set up instructors indicated that they felt as though they had to lecture 
and use that kind of instructional model. The immediacy of communication was also 
noticed with the lack of direct two way communication when using e-mail and Internet 
messages. Phone calls can help to bridge that gap but many times leaving messages can 
make the gap appear even larger. 

Using broadcast video reduced the amount of student to student interaction. In 
most cases no sites were used so there were no opportunities for informal 
communications. A few professors indicated that they had team projects in their course, 
but most instructors indicated that they didn’t know if the students ever interacted and 




14 



15 



06/29/98 



Assessment in Distance Learning 



they had not designed any interaction into the course. One instructor was experimenting 
with a list serve, but found that it did not produce much interaction. 

Internet 

Internet delivery was one of the smallest groups. It was difficult to tell in many 
cases whether the primary delivery system was Internet or text. Twelve courses feel into 
this category and the class size ranged from 5 students to 29. 

The biggest advantage of Internet delivery was the availability. The course 
content is available over the Internet 24 hours a day. Instructors can be reach by e-mail at 
any time. Several instructors indicated that they were always available for the students. 
This lead to one of the biggest disadvantages for the instructors, “I have no life.” Many 
instructors indicated that they felt tied to their computers and that they received phone 
calls, e-mails and faxes all the time. They also indicated that they spent a considerable 
amount of time teaching over the phone and they felt like they were teaching the same 
content time and time again. 

Student to student communication is being addressed through forums, small group 
work (on-line), and list serves. It was surprising how much of this communication 
occurred via computers. Students could very easily go through a semester and never hear 
their teacher or their fellow students voices even thought they could be interacting with 
each other on a daily basis. 

Text Delivery 

Text delivery of content was one of the most difficult areas to define. The typical 
course in this category follows a traditional correspondence course model. These course 
commonly mail the syllabus, learning resources, and text books to the student and the 
student turns in assignments over the Internet, by fax, or by e-mail. Class sizes were very 
difficult to define. Most correspondence course are considered to have a class size of one 
because students can begin when they want and there is no class setting. On the other 
hand there were classes in the survey that used text delivery and had between 100 and 
150 students using text as the primary method of delivering the content. 
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The advantages of text delivery are tied to individual learning. Text based 
delivery allows students to set their own pace, receive individualized 1 on 1 feedback, 
and improve their writing skills. The disadvantages is that there is no personal contact. 
Students don’t contact other students, rarely speak with the instructor, and commonly 
don’t develop relationships while working on the course materials. Instructors also 
mentioned that it was frustrating sharing answers with only one student knowing that they 
would probably have to have the same conversation with another student within a week. 

Instructors rely heavily on communication technology with this type of delivery. 
With text based learning fax and e-mail are the primary ways that students submit work 
to teachers. Mail is commonly too slow because by the time the student receives the 
feedback they have already completed another assignment and failed to incorporate the 
feedback. Feedback is commonly the most important way instructors communicate with 
students in this setting and they invest a lot of time and effort making extensive 
comments. This is probably why text based delivery showed the highest amount of time 
spent on assessment related activities. 

Video Tape 

Video Tape is very similar to broadcast delivery because commonly students are 
isolated from one another. The average class size for video tape delivery was 34 students. 
Class sizes ranged from 16 students to 40. 

The advantage of video tapes was that almost everyone had access to the 
technology and you could reach a broad audience. The disadvantages concerned the 
quality of communication. Instructors liked the convenience of e-mail, but recognized 
that phone conversations provide verbal cues which can help their instruction. The 
feedback indicate that communication was cumbersome with this delivery method. 

Phone calls were repetitive and time intensive, and the group chats didn’t seem to work. 

Student to student communications were limited. Were tapes were delivered to 
sites some small group communication took place, but in most situations there was little 
student interaction. E-mail was used in a few instances to bring students together on 
work teams or for discussions. 
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Are distance learning instructors spending more time with on or off 
campus students? 

At the onset of the phone interviews it became apparent that most of the 
instructors were dealing with on and off campus students. Some of these instructors 
would have both traditional and distance learning students in the same course where 
others would have one section of on campus students and another section delivered via 
distance learning. The participants were asked which group they spent more time with 
and the results indicate a solid “it depends”. There was an even split between more time 
with on campus students, more time with off campus students, and spent about the same 
amount of time with both. 

One instructor indicated that the wrong dividing line was being used, the real 
issue is the quality of the students. Several of the instructors who indicated they spent 
more time with the on-campus students indicated that the off campus students were much 
better and required less assistance: “The distance learning students are a ‘A grade to a full 
grade better than the on-campus students”. Instructors also indicated that the interaction 
was based on the students initiative and only those who had questions contacted them. 

Administration also seemed to be an issue when considering where time was 
spent. Much of the time with distance learning students was spent answering e-mails. 
Many of these e-mails concerned administrative issues. The amount of time spent with 
distance learning students may very widely based on the amount and type of support the 
instructor has in delivering the course content. 

How is cheating being addressed in the distance learning environment? 

Cheating was the most emotionally charged issue covered in this research. There 
were several issues that came out when instructors were asked what they do when a 
student is caught cheating. Many instructors felt it was an issue at the master’s level, 
some responded to how difficult it was to prove that a student was cheating, and others 
shared their policies on cheating. 
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Several instructors use assessment which require creative thought to complete the 
assignment. Many times students are encouraged to use any resources, or people, to 
complete the assignment. In this case it is difficult to define what cheating is. An area of 
concern that did arise was student buying papers. Different cultures view cheating 
differently. One instructor indicated that foreign students have purchased papers and 
submitted them thinking that it was acceptable because they had paid for them. The 
definition of cheating was also questioned when students would submit papers they had 
submitted to previous classes. 

Many instructors shared their fhistration with cheating in general. It is very hard 
to define what cheating is and the burden of proof rests on the professor. Several 
professors indicated that it just wasn’t worth while to pursue the official review process. 
Instructors indicated that if the student wouldn’t confess they wouldn’t pursue the issue. 

Because of the fhistration, some instructors have developed their own policies on 
cheating. These policies ranged from receiving a zero on the assignment to being thrown 
out of the class. It was disappointing to see that only 15% of the syllabi collect had 
academic dishonesty policies in them. On one extreme. Regent’s University has a 
standard phrase for how academic dishonesty will be addressed: “Transgressions shall be 
handled according to John 7:51.” Most instructors indicated that the first step was to 
speak with the student and then see if a solution could be identified. 

There are several processes that are being used to help reduce the possibility of 
cheating. One that was mentioned earlier was keeping the course current. When 
assignments are current there is no previous student work done in that area and the 
students don’t have the option to cheat. Many instructors using 2 way video rely heavily 
on site coordinators to help them monitor student during assessment. It is becoming 
more common for those taking text based course to be required to set up a proctor at the 
beginning of the course which will be used to ensure honesty. Acceptable proctors 
include professors, librarians, and professional testing centers. 
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Conclusion 

Distance learning is still a growing field full of experimentation and exploration. 

It was apparent through this research that instructors are not sharing their ideas, solutions, 
and failures with other professors in their own institutions or in other institutions. This 
has lead to many individuals having to recreate the wheel. 

This research found assessment practices in general to be consistent with 
traditional education. Distance learning instructors haven’t created new types of 
assessments and they are choosing assessments based on their needs instead of letting the 
technology dictate what they will use. 

Distance learning instructors could benefit from more support from their 
institutions in the area of quality control. Many assessment practices are being adversely 
affected by university policy and instructors indicated that they were not getting the 
support they needed. Some of the emerging distance learning models are exciting 
because instructors are well supported and some good steps have been taken to ensure 
quality across instructors. Some of these steps should be adopted by more traditional 
institutions to help them prepare for the future. 



Appendix A 

How was this information collected and who participated? 

Participants were selected based on course listings foimd in Peterson’s Guide to 
distance learning. 25 imiversity offering an MBA degree in the Fall of 1997 was solicited 
to participate, of those 21 imiversities had professors who completed the e-mail survey, 
phone interview, and provided a copy of their syllabus. The total number of participants 
was 62. 

There were 10 categories of courses being taught: accoimting, economics, ethics, 
finance, international business, (management), human resource management, quality 
management, regular management, and marketing. Management was one of the largest 
categories so it was broken down into sub categories. Class size varied widely among the 
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different categories. International business was only taught to large classes (45 or more 
students) and ethics was only taught to small or medium sized classes. 

The surveys used to collect the data are included in the appendices. Appendix C 
is the original e-mail survey. This survey was sent after the professor had indicated that 
they were willing to participate. Appendix B is the interview guide used in the follow up 
phone interview. 
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Appendix B 

Assessment Strategy Interview Guide 

Professors Name: Date: 

Survey Questions Time: 

1. How did you determine the assessment components of your course (exams, quiz)? 



a) Did you use any university policies or guidelines? 



b) What criteria did you use when selecting different tools? 



c) How did you determine grade cutoffs? 

2. How did you determine your assessment strategy? What factors played into your 
decision? 



O 

ERIC 



a) Was the class size a factor? 



b) Was the way you communicated with students a factor? 



c) Did the technology have an impact on your assessment strategy? 



d) Was security an issue in determining your assessment strategy? 



e) Was the time necessary to grade or administer assessments and issue? 



f) Was the evaluation of the assessment an issue? 
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g) Did you get any advice from anyone with more experience? 

3. Special grading situations 

a) How do you decide borderline grades? What factors influence your decision? 

b) How do you deal with a student who hasn’t completed an assessment? 

c) What do you do if a student is caught cheating? 

d) Can students recover from a bad first assessment? 

4. Have you done anything to check the quality of your assessments? (validity, 
reliability, generalizability) 

5. How do your assessments impact the students learning? 



6. What are the advantages and disadvantages of the assessments you use? 



7. What are the advantages and disadvantages of the way you communicate with the 
students? 
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Appendix C 

Dear «Professor»: 

Your willingness to participate in the distance learning assessment 
research is greatly appreciated. 1 recognize that you are very busy, so 1 am 
resending the survey because your input is critical to the success of this project. 
Your response will be significant in improving the study and my understanding 
of how assessment is handled at «University». The following survey is designed 
to collect information about the assessment methods you are using in 
«Course_Name». Information concerning how to reply (by e-mail, fax, or snail 
mail) is included at the end of the survey. 

Thardc you for your help. 

Matt Dirks 



I. Demographics (Pre-Interview research) 

(This section was filled out automatically. Please check to ensure that all of the 
information is accurate and complete.) 



1 . The Course Number: 



«COURSE NUMBER» 



2. The Course Name: 



«COURSE NAME» 



3. The Instractor's Name: 



«Professor» 



4. The Instmctor's E-mail Address: 



«EMAIL ADDRESS» 



5. The Instmctor's Office Phone Number: 



«Phone» 



6. The institution where the course is taught: «University» 



II. Course Information 

(Please answer each of the following questions for the course listed above.) 
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1 . How long have you been teaching college courses? 

2. How long have you been teaching this course? 

3. How many college courses have you taught via distance learning including this 
semester? 

4. How many students are in the class this semester (include all sites)? 

5. If you were using distance sites, how many sites were involved? 

6. How much help did you have teaching this course? (who are they?) 

(e.g.: graduate teaching assistant, site coordinators, web master, etc.) 

7. At the beginning of the course did you communicate the course objectives (or 
goals) to the students? If so, how? 

8. At the beginning of the course did you communicate the grading system to the 
students? If so, how? 

9. How is the course content being delivered to learners? (video, Internet, text, etc.) 

10. Considering all the time you spend on this course (100%), how much of that time 
do you think you spend on student assessment (writing tests, grading, giving 
feedback, and reporting scores)? 

III. Assessment Information 

1 . In this course, what purpose do your assessments serve? (why do you have them?) 

2. Please complete the table below indicating what you base grades on? (two 
examples provided) 

a) name or title of each assessment 

b) relative weight of each assessment (points or percentage of the final grade) 

c) description of the frequency and length of each assessment 

(number of tests and the number of questions on each test) 

d) description of the evaluation of the assessment (how it was scored) 



Example #1 

Assessment Weight Frequency/length Evaluation 
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1. Tests 


40% 


2, 1 00 questions 


Computer graded 


2. Papers 


45% 


1 , 20 pages 


Instructor scored, w/ comments 


3. Participation 


15% 


1 0, attendance 


Instructor scored based on homework 


Example #2 


Assessment 


Weight 


F requency/length 


Evaluation 


1. Quizzes 


200 pts. 


4 quizzes, 1 0 questions 


TA scored 


2. Test 


200 pts. 


2, 1 00 questions 


Instructor scored 


3. Homework 


300 pts. 


10, 5-10 questions 


Instructor scored with comments 


4. Attendance 


100 pts. 


1 0 attendance checks 


TA checked in lab 



Assessment Weight Frequencv/Length Evaluation 

1 . 

2 . 

3. 

4. 

5. 

6 . 



3. Are there other assessments (or assignments) that you used that are not included in the 

grade? If so, what are they? 

1 . 

2 . 

3. 

IV. Feedback and Student Interaction 

1 . How do students submit their work? (e-mail, fax, in person, to a site coordinator) 

2. How do students find out how they did on your assessments? 

3. When students have questions, what are the most common ways they contact you? 

4. When you receive student questions, what are the most common ways you respond? 
(e-mail, phone, during lectures) 
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5. What are the most common things students ask about? (content issues, technical 
questions, clarify issues, grading issues) 

6. How do students interact with other students (peer to peer interactions)? 

V. Follow Up 

(You have completed the time consuming part of this research. There are only 
three things left: the phone interview, the syllabus, and the course home page.) 

1 . When are good times to call you? 

2. Is this phone number, «Phone», the best one to use to contact you? 

3. How can I get a copy of your syllabus? (on the web, e-mail, fax, mail) 

4. If there is a course home page can I look at it? (What is the URL and are there any 
user names or passwords that I will need?) 



Thank you for your participation. Your cooperation is invaluable to furthering 
research in distance learning. 



Matthew Dirks 

home: (801)802-7257 1385 N. Amiron Way #A 

fax: (801)496-4156 Orem, UT 84057 

med43 3 3 @gamet.acns .fsu.edu 



Matt Dirks is in the Instructional Systems program at Florida State University. This survey is part of his 
dissertation research efforts. 
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